Method and apparatus for processing a 3d scene

ABSTRACT

A method and an apparatus for processing a 3D scene are disclosed. At least one virtual reference viewpoint in the 3D scene is determined (41). A map of registered real cast shadows of objects in the 3D scene from an input image captured by a camera positioned at a viewpoint distinct from the virtual reference viewpoint is obtained (42), said map of real cast shadows being registered with regards to the virtual reference viewpoint. Parameters for at least one light source in the 3D scene are determined (44) using the map of registered real cast shadows and at least one map of virtual shadows of objects in the 3D scene cast by the at least one light source from the virtual reference viewpoint.

1. TECHNICAL FIELD

The present disclosure relates to 3D scene lighting for mixed reality.More particularly, the present disclosure relates to virtual objectlighting inserted into a 3D real scene.

2. BACKGROUND

In mixed reality, that is when virtual objects are inserted into a 3Dmodel of a real scene, 3D lighting of the virtual objects is a keyfeature in order to provide a realistic aspect of the 3D scene. Thevirtual objects should be lighted correctly by imitating the reallighting of the scene. However, lighting environment may changedynamically. Therefore, 3D lighting of a real scene has to becontinuously estimated.

But lighting is a time-consuming task in real-time rendering and needsto be drastically simplified in particular when rendered on a mobiledevice. Therefore, a compromise must be found between complex modelingand graphics rendering time.

Therefore, there is a need for a new method that can model complexindoor dynamic scene light sources.

3. SUMMARY

According to an aspect of the present disclosure, a method forprocessing a 3D scene is disclosed. Such a method comprises determiningat least one virtual reference viewpoint in the 3D scene, obtaining amap of registered real cast shadows of objects in the 3D scene from aninput image captured by a camera positioned at a viewpoint distinct fromthe virtual reference viewpoint, said map of real cast shadows beingregistered with regards to the virtual reference viewpoint, determiningparameters for at least one light source in the 3D scene using the mapof registered real cast shadows and at least one map of virtual shadowsof objects in the 3D scene cast by the at least one light source fromthe virtual reference viewpoint.

According to another aspect of the present disclosure, an apparatus forprocessing a 3D scene is also disclosed. Such an apparatus comprisesmeans for determining at least one virtual reference viewpoint in the 3Dscene, means for obtaining a map of registered real cast shadows ofobjects in the 3D scene from an input image captured by the camerapositioned at a viewpoint distinct from the virtual reference viewpoint,said map of real cast shadows being registered with regards to thevirtual reference viewpoint, means for determining parameters for atleast one light source in the 3D scene using the map of registered realcast shadows and at least one map of virtual shadows of objects in the3D scene cast by the at least one light source from the virtualreference viewpoint.

4. BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary method for determining 3D position ofcandidates point light sources of the 3D scene according to anembodiment of the present disclosure,

FIG. 2A illustrates an image of a 3D scene,

FIG. 2B illustrates cast shadows of objects in the 3D scene from theimage illustrated by FIG. 2A,

FIG. 2C illustrates an image of the 3D scene rendered with a point lightsource,

FIG. 2D illustrates another image of the 3D scene rendered with anotherpoint light source,

FIG. 3A illustrates different viewpoints for capturing a real scene froma camera embedded in mobile device,

FIG. 3B illustrates virtual reference viewpoints introduced in the 3Dscene,

FIG. 4 illustrates an exemplary method for processing a 3D sceneaccording to an embodiment of the present disclosure,

FIG. 5A illustrates an exemplary method for obtaining a map ofregistered real cast shadows according to an embodiment of the presentdisclosure,

FIG. 5B illustrates an exemplary method for obtaining a map ofregistered real cast shadows according to another embodiment of thepresent disclosure,

FIG. 6A illustrates an exemplary method for determining a virtualreference viewpoint according to an embodiment of the presentdisclosure,

FIG. 6B illustrates an exemplary method for determining a virtualreference viewpoint according to another embodiment of the presentdisclosure,

FIG. 7 illustrates an exemplary method for lighting parametersdetermination according to an embodiment of the present disclosure,

FIG. 8 illustrates an exemplary apparatus for processing a 3D sceneaccording to an embodiment of the present disclosure.

5. DESCRIPTION OF EMBODIMENTS

In order to correctly light a virtual scene, rendering correctly litvirtual objects and faithfully rendering shadows cast by the virtualobjects onto the real scene is essential. Two main features contributeto the realism of the shading and of the cast shadows: the 3Dlocation/orientation of the shadows and their strength given by theattenuation of the color intensity of the shaded surface.

In presence of hard shadows, a set of virtual lighting models exists:point lights, spot lights or directional lighting.

In the embodiments disclosed herein, lighting is modeled by a set of 3Dpoint light sources. The 3D location of such source light points and theshadow attenuation associated to each source light point can beestimated from a unique video camera observing the real scene.

It should be noted that lighting conditions in a real scene are notstable and change continuously. Therefore, for improving the realism ofmixed reality, lighting environment has to be estimated continuously.

In order to estimate the 3D location of the light sources in the realscene, the real scene is analyzed via image processing.

FIG. 1 illustrates an exemplary method for light sources locationdetermination.

It is assumed that a 3D geometric model of the real scene is availableas well as an image of the scene captured from an input video camerawith known 3D pose in the 3D model coordinate system. The 3D pose of theinput video camera not only comprises the 3D location of the input videocamera in the 3D model coordinate system, but also intrinsic cameraparameters.

For instance, location in the 3D scene of point light sources isdetermined for point light sources from a set of candidate point lightsources. Such a set of 3D point light sources can be a list of 3D posesor a structured tree of 3D locations.

In step 10, real cast shadows are detected in an input RGB (Red GreenBlue) image of the scene obtained for the input video camera. Real castshadows correspond to shadows of the real objects present in the scenecast by the real light sources and captured by the input video camera.

FIG. 2A illustrates an example of an input RGB image. FIG. 2Billustrates corresponding cast shadows detected in the input image. OnFIG. 2B, detected cast shadows appear in light grey and are identifiedby the arrows pointing from the reference 20.

In step 11, for a candidate light source, the virtual shadow obtained bythe candidate light source are matched with the detected cast shadows.

In order to do so, in step 11, a rendered image is obtained for thecandidate point light source. Such a rendered image may be obtained byany known method for 3D rendering cast shadows from the candidate pointlight source. The rendered image comprises shadows cast by theassociated candidate point light source from the set of point lightsources.

FIGS. 2C and 2D illustrates examples of images rendered with differentpoint light sources, and using the viewpoint in the 3D scene as theviewpoint of the input video camera, similar as the viewpoint used forobtaining the input RGB image.

It can be seen on FIGS. 2C and 2D that cast shadows, respectively 21 and22, depend on the location of the 3D point light source used to renderthe image. On FIG. 2C, the 3D point light source is located on the leftof the projected scene, while on FIG. 2D, the 3D point light source islocated on the right of the projected scene.

Then, in step 11, the rendered image is matched with the mask of realcast shadows detected at step 10. Matching the rendered image with thedetected real cast shadow may be carried out via the computation ofcorrelation between the binary mask of the detected real cast shadowsand the binary mask of the virtual shadows cast by the candidate 3Dpoint light source.

The 3D position of the point light source corresponds to the 3D locationof the candidate point light source.

In step 11, once the location of a candidate point light source isdetermined, matched pixels that match both the mask of detected realcast shadows and the virtual cast shadows obtained from the candidatepoint light source are marked so as to discard those pixels whenestimating other point light sources location.

When estimating other point light sources, in step 11, the pixels thathave been marked in determining previous point lights are discarded.

In step 12, it is verified if a maximum number of point light sourceshave not yet been determined, and if the number of unmatched real shadowpoints is above a determined value. For instance, a maximum of pointlight sources may correspond to a maximum number of point light sourcesthat can be satisfactorily processed by the rendering engine.

If both conditions cited above are true, one more point light source isselected from the set of candidates 3D point light sources and added tothe 3D scene. The matching operation is repeated for the new 3D pointlight source (step 11). It should be noted that when repeating thematching operation for a new 3D point light source, the detected realshadow points that match virtual shadow points cast from a 3D pointlight source previously determined are discarded.

When at least one condition among the two conditions cited above isfalse, 3D point light sources location determination ends, and in step13, an occlusion attenuation is computed and assigned to each 3D pointlight source determined in step 11.

From the exemplary method discussed above, it can be seen that a virtualshadow map, i.e. a map of shadows cast by a candidate 3D light sourcepoint, has to be computed for each candidate source light. Thus, thecreation of the set of virtual shadow maps is expensive.

Moreover, these maps are associated to a particular viewpoint thatcorresponds to the input video camera viewpoint that captures an imageof the real scene. If the lighting environment is stationary and if thegeometry of the scene is static, the lighting determination discussedabove can be carried out off-line. But on the other hand, if thelighting environment is dynamic, the lighting determination must be donecontinuously on-line.

Furthermore, light sources are determined depending on the input videocamera viewpoint. When the input video camera viewpoint changes, castshadows, real and virtual, change.

Such a problem may appear, for instance, when the 3D scene processingdiscussed above is used for mixed reality applications provided onmobile devices, such as mobile phones, tablets, etc. . . . . In suchapplications, the real scene may be captured by the video camera of themobile device. The user of the mobile device may move the mobile device,and thus the input video camera viewpoint may change. For instance, asillustrated on FIG. 3A, the input video camera may capture the realscene from a first viewpoint V1. The user then moves with the mobiledevice, on the right, towards the viewpoint V2, the real scene is thencontinuously captured from the successive viewpoints along the inputvideo camera trajectory.

In that case, as light sources have to be continuously determined andthe viewpoint of the input video camera changes, the set of virtualshadow maps must be continuously updated. However, the computation ofnumerous shadow maps is time-consuming. So, an objective of the presentdisclosure is to limit the occurrence of shadow maps computation.

According to the principles presented here, virtual reference viewpointsare introduced in the 3D scene. The set of virtual shadow maps is thuscreated with respect to the virtual reference viewpoints. As long as theinput video camera remains close to a virtual reference viewpoint, thevirtual shadow maps created in regards with this virtual referenceviewpoint are used in the lighting parameters determination process.

In the particular case where the moving camera locations are roughlyknown, a set of virtual reference viewpoints can be pre-defined and thecorresponding shadow maps can be pre-computed. On the other hand, if thecamera motion cannot be controlled, the virtual reference viewpointsmust be defined on the fly.

In this latter case, when the input video camera moves and reaches aposition where the current virtual reference viewpoint is no more valid,a new virtual reference viewpoint is defined and a new set of virtualshadow maps is created. The new virtual reference viewpoint may beselected for instance as the new location of the input video camera.

FIG. 3B illustrates exemplary locations V1, V3, V4, V2, for virtualreference viewpoints introduced in the 3D scene.

According to an embodiment of the present principle, the virtualreference viewpoints may correspond to viewpoints of a multi-view imagecapture. In this way, camera parameters and location are already known.

According to the principles presented here, lighting parametersdetermination (step 11 from FIG. 1) is modified as follows. FIG. 4illustrates an exemplary method for lighting parameters determinationaccording to an embodiment of the present disclosure.

The real scene is captured by a camera from a first viewpoint, whichwill be called the input video camera viewpoint in the following. Forinstance the mobile device is located at the viewpoint V1′ in the 3Dscene as illustrated on FIG. 3B.

During the 3D scene rendering process, virtual cast shadows maps aredetermined with respect to a current virtual reference viewpoint. At thebeginning of the rendering process, the current virtual referenceviewpoint corresponds to the input video camera viewpoint as the inputvideo camera may have not yet been moved by the user. Virtual castshadows maps are determined as discussed above in reference with FIG. 1.

In a particular case, it may be asked the user to begin at a roughlocation from which a pre-processed reference viewpoint can be used withits pre-computed shadow maps.

In step 40, it is checked whether a new virtual reference viewpoint isneeded for lighting parameters determination.

According to an embodiment of the present disclosure, a new virtualreference viewpoint is needed if the viewpoint of the input video cameraand the current virtual reference viewpoint do not satisfy a determinedcriterion. For instance, a new virtual reference viewpoint is determinedif a number of pixels overlapping in an image captured from the currentvirtual reference viewpoint and an image captured from the currentviewpoint of the input video camera is below a determined value. As anexample, the pixels are declared “overlapping” if after registration ofone of the two images on the other, the distance between two co-locatedpixels in terms of absolute difference of depth is below a threshold.

According to another example, a new virtual reference viewpoint isdetermined if a number of pixels from an image captured from the currentvirtual reference viewpoint having no corresponding point in the imagecaptured from the current viewpoint of the input video camera is above adetermined value.

According to another embodiment of the present disclosure, a new virtualreference viewpoint is needed if the distance between the input videocamera viewpoint and the current virtual reference viewpoint is above adetermined value. Such a determined value may depend on the complexityof the real scene, taking into account for instance the number ofobjects in the scene, the number and locations of the 3D light sources,etc. . . . .

If a new virtual reference viewpoint is needed, at step 41, a newvirtual reference viewpoint is determined and the underlying candidateshadow maps are obtained.

If the new virtual reference viewpoint has been previously used or ifthe new virtual reference viewpoint has been previously determined andadded to a set of available virtual reference viewpoints, its underlyingcandidate shadow maps may have been previously computed and stored. Inthat case, the candidate shadow maps associated to the new virtualreference viewpoint are retrieved from memory.

If the candidate shadow maps associated to the new virtual referenceviewpoint have not yet been computed, they are computed for instance asdiscussed above with FIG. 1. A candidate shadow map is thus obtained foreach of the candidate point light source using any known method for 3Drendering cast shadows.

For instance, the new virtual reference viewpoint is selected among aset of candidates viewpoints whose location is known. In the example ofFIG. 3B, the set may comprise the viewpoints V1-V4.

According to an embodiment of the present disclosure, the virtualreference viewpoint among the set of candidates viewpoints is selectedas the closest viewpoint from the current input video camera viewpoint.

According to another embodiment of the present disclosure, the virtualreference viewpoint among the set of candidates viewpoints is selectedas a viewpoint for which a number of pixels overlapping in an imagecaptured from the candidate viewpoint and the image captured from thecurrent input video camera viewpoint is above a determined value. Theselected viewpoint may be the candidate viewpoint having the most pixelsoverlapping with the input video camera viewpoint.

It has been assumed above that a set of virtual reference viewpoints isavailable among which a satisfying reference viewpoint can be identifiedwhatever the current input video camera viewpoint is. This can be thecase if the space of locations and orientations of the input videocamera viewpoints is a priori known. Otherwise, if for example thecamera motion cannot be controlled, the reference viewpoint and theunderlying candidate shadow maps are defined on the fly. FIG. 6Bdiscussed below discloses an exemplary method for determining virtualreference viewpoints on the fly, according to an embodiment of thepresent disclosure.

In step 42, a new map of registered real cast shadows is obtained. Thereal cast shadows are the shadows cast by the real light sources fromthe real objects in the real scene. The real cast shadow map is obtainedfrom an input image captured by the input video camera at the currenttime instant. As the current viewpoint of the input video camera may bedistinct from the new virtual reference viewpoint, the real cast shadowmap has to be registered with regards to the new virtual referenceviewpoint.

The current map of registered real cast shadows is continuously updateddue to the fact that lighting may have changed or real camera may havemoved, and consequently cast shadows detected in the current realviewpoint may have changed.

In step 44, lighting parameters are determined using the currentregistered real cast shadow map and maps of virtual shadows cast by 3Dpoint light sources and associated to the current virtual referenceviewpoint. A map of virtual shadows is generated for each candidate 3Dpoint light source, with respect to the current virtual referenceviewpoint.

Lighting parameters are determined as discussed above in relation withFIG. 1, each map of virtual shadows is matched with the map ofregistered real cast shadows obtained at step 42. And the map of virtualshadows that best matches the map of registered real cast shadows isselected. The 3D point light source associated to the selected map ofvirtual shadows is selected as a 3D point light source. This step isiterated with other maps of virtual shadows for selecting other 3D pointlight sources. Pixels from the map of registered real cast shadows thathave been marked as matching a previous map of virtual shadows arediscarded in the matching process for the subsequent 3D point lightsource determination.

FIG. 5A illustrates an exemplary method for obtaining a map ofregistered real cast shadows according to an embodiment of the presentdisclosure.

According to this embodiment, in step 50, the real cast shadows aredetected in the image captured from the input video camera, in a similarmanner as discussed above with FIG. 1. A map of detected real castshadows is thus obtained. Such a map reflects the real cast shadows fromthe input video camera viewpoint.

In step 51, the map of detected real cast shadows is registered withregards to the current virtual reference viewpoint. Such a registrationis carried out by determining for each pixel of the map of detected realcast shadows, a corresponding pixel or pixels in the registered realcast shadows map. This can be done by determining the 3D point in the 3Dscene that corresponds to the pixel projected in map of detected realcast shadows. The determined 3D point in the 3D scene is then projectedin the current virtual reference viewpoint. The pixel in the registeredreal cast shadows map corresponding to the pixel in the map of detectedreal cast shadows is set to the same binary value, for instance 0 if thepixel is a detected shadow pixel, 1 if the pixel is not a detectedshadow pixel.

FIG. 5B illustrates an exemplary method for obtaining a map ofregistered real cast shadows according to another embodiment of thepresent disclosure.

According to this embodiment, in step 52, the image captured by theinput video camera is registered with regards to the current virtualreference viewpoint. This can be carried out in a same way as discussedabove, by projecting the pixels in the 3D space and then projecting the3D points with regards to the current virtual reference viewpoint. Aregistered input image is thus obtained.

In step 53, real cast shadows are then detected in the registered inputimage obtained at step 52.

FIG. 6A illustrates an exemplary method for determining a virtualreference viewpoint according to an embodiment of the presentdisclosure.

According to this embodiment, a new virtual reference viewpoint may bedetermined in advance before this new virtual reference viewpoint isneeded for lighting parameters determination. According to thisembodiment, the camera trajectory may be predicted. A new virtualreference viewpoint is determined in advance according to the trajectoryof the camera and virtual shadows map can be determined in advanceaccording to this new virtual reference viewpoint. In that way, when thecurrent virtual reference viewpoint is no more valid, the new virtualreference viewpoint determined in advance can be used.

According to this embodiment, in step 60, movement of the cameracapturing the real scene is detected.

In step 61, the camera trajectory is predicted by extrapolating themovement of the camera.

In step 62, the new virtual reference viewpoint is determined accordingto the predicted camera trajectory. A multi-view viewpoint on the cameratrajectory may be selected or a viewpoint close to the cameratrajectory.

Virtual shadows maps can then be determined and stored for future use.

FIG. 6B illustrates an exemplary method for determining a virtualreference viewpoint according to another embodiment of the presentdisclosure. According to this embodiment, virtual reference viewpointsare determined on the fly, to anticipate the fact that a new virtualreference viewpoint may be needed by the system disclosed with FIG. 4.

A new reference viewpoint can be defined as one of the last real inputvideo camera viewpoints or the current real input video camera viewpointor a virtual viewpoint corresponding for example to an average locationand orientation of the last real input video camera viewpoints. As thecomputation of the corresponding virtual shadow maps is time-consuming,anticipation is required.

In step 63, camera pose is tracked continuously.

In step 64, it is checked whether the camera is moving away from thecurrent reference viewpoint from more than a threshold. For instance, adistance between the current virtual reference viewpoint and thecamera's viewpoint is determined, and it is checked whether thisdistance is above a determined value. According to a variant, it couldbe checked if a number of pixels overlapping in an image captured fromthe current virtual reference viewpoint and an image captured from thecamera's viewpoint is below a determined value. In that case, it isconsidered that the camera has moved away from the current referenceviewpoint from more than the threshold. Such overlapping measurements onsuccessive viewpoints can allow to evaluate the trend and predict theevolution of the overlapping measure in order to take a better decisionwith respect to the possible introduction of a new reference viewpoint.

If this is the case, in step 65, a new reference viewpoint isdetermined, for example at the current real camera pose. The virtualshadow maps associated to the new reference viewpoint are computed.

While this new reference viewpoint is being defined, the currentreference virtual viewpoint is still used for lighting parametersdetermination. So, the threshold used at step 64 is chosen so thatbeyond it, it is still possible to use the current reference viewpointwhile the virtual shadow maps of the new reference viewpoint are beingcomputed.

When the new reference viewpoint is ready, at step 66, the two referenceviewpoints (current reference viewpoint and new reference viewpoint)poses are compared with the current real camera pose and the referenceviewpoint that is closest to the current real camera pose is selected.The selected reference viewpoint is then used in the process forlighting parameters determination as disclosed with FIG. 4.

At step 67, the reference virtual viewpoints and their correspondingshadow maps can also be stored to be used later, so that they enrich theset of available reference virtual viewpoints. In this context, theselection is mixed: either a reference virtual viewpoint is selectedamong available ones if a satisfying one exists or a new one is computedand then stored.

The virtual reference viewpoint may be selected to be close to the areawhere the video camera moves, in order to minimize the effect of theerror on the 3D position of the virtual reference viewpoint andconsequently the errors on the registered real shadow map. It can be forexample a real viewpoint selected once it is captured. The selectedvirtual viewpoint can then be kept as long as for example theoverlapping between the real and virtual viewpoint images is more than apercentage of pixels.

A condition to change the virtual reference viewpoint is that the systemhad time to generate new virtual shadow maps corresponding to the nextvirtual reference viewpoint.

The creation of the virtual maps for a new virtual reference viewpointcan also be carried out in a coarse to fine mode by guaranteeing firstthat the 3D space is covered with distant light positions and by thenprogressively introducing new light positions. In this mode, once a newvirtual viewpoint is selected, the determination of lighting parametersis first carried out with a subset of virtual shadow maps, leadingpossibly to a coarse estimate of the light positions. Then, as newvirtual shadow maps are created, the estimation can be more accurate.

In the description above, it has been assumed that the geometry of theobserved 3D scene is not changing significantly. On the other hand, if apart of the scene is modified, e.g. an object is moved, inserted orremoved, scene geometry changes should be detected and taken intoaccount for the computation of the shadow maps associated to a virtualreference viewpoint and the lighting parameters determination.

FIG. 7 illustrates an exemplary method for lighting parametersdetermination according to an embodiment of the present disclosure whengeometry of the scene changes.

In step 70, geometry changes in the scene are detected. For instance,the current geometry of the scene is captured and compared with thecurrently stored 3D model.

If there is no significant change in the geometry, at step 75, lightingparameters determination is performed using the method discussed withFIG. 4. By significant, it should be understood here, a geometry changethat has an impact on the virtual cast shadows associated to the currentvirtual reference viewpoint.

If there is a significant change in the geometry, at step 71, the new 3Dmodel and the currently stored 3D model are projected onto the currentviewpoint. The 2D areas that are impacted by the modification in the 3Dscene are identified. These 2D areas are discarded in the input imagesof the viewpoint, for instance by defining a mask identifying thediscarded areas

In step 72, lighting parameters determination is performed using themethod discussed with FIG. 4, using the masks so that the discarded 2Dareas are not analyzed in the input images. Meanwhile, in step 73, thegeometric model is updated.

Once the geometric model has been updated, in step 74, new virtualshadow maps are computed in the virtual reference viewpoint.

Once the new virtual shadow maps are ready, the 3D model is updated forall the tasks (including processing of the input frames), the updatedvirtual reference viewpoint is used, and the old stored virtualreference viewpoints are removed. Thus, the management of virtualreference viewpoints is re-initialized.

Then, at step 75, lighting parameters determination is performed usingthe method discussed with FIG. 4, without the masks defined at step 71as the management of virtual reference viewpoints has beenre-initialized.

FIG. 8 illustrates an exemplary apparatus for processing a 3D sceneaccording to an embodiment of the present disclosure. FIG. 8 illustratesa block diagram of an exemplary system 800 in which various aspects ofthe exemplary embodiments may be implemented. The system 800 may beembodied as a device including the various components described belowand is configured to perform the processes described above. Examples ofsuch devices, include, but are not limited to, mobile devices, personalcomputers, laptop computers, smartphones, tablet computers, digitalmultimedia set top boxes, digital television receivers, personal videorecording systems, connected home appliances, and servers. The system800 may be communicatively coupled to other similar systems, and to adisplay via a communication channel.

Various embodiments of the system 800 include at least one processor 810configured to execute instructions loaded therein for implementing thevarious processes as discussed above. The processor 810 may includeembedded memory, input output interface, and various other circuitriesas known in the art. The system 800 may also include at least one memory820 (e.g., a volatile memory device, a non-volatile memory device). Thesystem 800 may additionally include a storage device 840, which mayinclude non-volatile memory, including, but not limited to, EEPROM, ROM,PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and/or optical diskdrive. The storage device 840 may comprise an internal storage device,an attached storage device, and/or a network accessible storage device,as non-limiting examples.

Program code to be loaded onto one or more processors 810 to perform thevarious processes described hereinabove may be stored in the storagedevice 840 and subsequently loaded onto the memory 820 for execution bythe processors 810. In accordance with the exemplary embodiments, one ormore of the processor(s) 810, the memory 820, and the storage device840, may store one or more of the various items during the performanceof the processes discussed herein above, including, but not limited toreal and virtual cast shadows map, input image, 3D scene geometry,virtual reference viewpoint's 3D pose, variables, operations, andoperational logic.

The system 800 may also include a communication interface 850 thatenables communication with other devices via a communication channel.The communication interface 850 may include, but is not limited to atransceiver configured to transmit and receive data from thecommunication channel. The communication interface 850 may include, butis not limited to, a modem or network card and the communication channel850 may be implemented within a wired and/or wireless medium. Thevarious components of the system 800 may be connected or communicativelycoupled together (not shown) using various suitable connections,including, but not limited to internal buses, wires, and printed circuitboards.

The system 800 also includes user interactions means 830 coupled to theprocessor for receiving user inputs.

The system 800 also includes video capturing means 860, such as acamera, coupled to the processor for capturing video images.

The exemplary embodiments may be carried out by computer softwareimplemented by the processor 810 or by hardware, or by a combination ofhardware and software. As a non-limiting example, the exemplaryembodiments may be implemented by one or more integrated circuits. Thememory 820 may be of any type appropriate to the technical environmentand may be implemented using any appropriate data storage technology,such as optical memory devices, magnetic memory devices,semiconductor-based memory devices, fixed memory, and removable memory,as non-limiting examples. The processor 810 may be of any typeappropriate to the technical environment, and may encompass one or moreof microprocessors, general purpose computers, special purposecomputers, and processors based on a multi-core architecture, asnon-limiting examples.

The implementations described herein may be implemented in, for example,a method or a process, an apparatus, a software program, a data stream,or a signal. Even if only discussed in the context of a single form ofimplementation (for example, discussed only as a method), theimplementation of features discussed may also be implemented in otherforms (for example, an apparatus or a program). An apparatus may beimplemented in, for example, appropriate hardware, software, andfirmware. The methods may be implemented in, for example, an apparatussuch as, for example, a processor, which refers to processing devices ingeneral, including, for example, a computer, a microprocessor, anintegrated circuit, or a programmable logic device. Processors alsoinclude communication devices, such as, for example, computers, cellphones, portable/personal digital assistants (“PDAs”), tablets, andother devices that facilitate communication of information betweenend-users.

A method for processing a 3D scene is disclosed. Such a method comprisesdetermining at least one virtual reference viewpoint in the 3D scene,obtaining a map of registered real cast shadows of objects in the 3Dscene from an input image captured by a camera positioned at a viewpointdistinct from the virtual reference viewpoint, said map of real castshadows being registered with regards to the virtual referenceviewpoint, determining parameters for at least one light source in the3D scene using the map of registered real cast shadows and at least onemap of virtual shadows of objects in the 3D scene cast by the at leastone light source from the virtual reference viewpoint.

According to the present principle, a virtual reference viewpoint isdetermined for determining lighting parameters. The virtual referenceviewpoint may be selected to be close to the area where the video cameramoves, in order to minimize the effect of the error on the 3D positionof the virtual reference viewpoint and consequently the errors on theregistered real shadow map. It can be for example a real viewpointselected once it is captured.

The selected virtual viewpoint can then be kept as long as for examplethe overlapping between the camera's viewpoint and virtual viewpointimages is more than a percentage of pixels. Thus, on-line lightingparameters determination complexity is reduced, online 3 d renderingprocessing is possible on mobile devices.

An apparatus for processing a 3D scene is also disclosed. Such anapparatus comprises means for determining at least one virtual referenceviewpoint in the 3D scene, means for obtaining a map of registered realcast shadows of objects in the 3D scene from an input image captured bythe camera positioned at a viewpoint distinct from the virtual referenceviewpoint, said map of real cast shadows being registered with regardsto the virtual reference viewpoint, means for determining parameters forat least one light source in the 3D scene using the map of registeredreal cast shadows and at least one map of virtual shadows of objects inthe 3D scene cast by the at least one light source from the virtualreference viewpoint.

According to an embodiment of the present disclosure, obtaining the mapof registered real cast shadows comprises detecting the real castshadows in the input image to obtain a map of detected real castshadows, and registering the map of detected real cast shadows withregards to the virtual reference viewpoint.

According to an embodiment of the present disclosure, obtaining the mapof registered real cast shadows comprises registering the input imagewith regards to the virtual reference viewpoint to obtain a registeredimage, and detecting the real cast shadows in the registered image.

According to an embodiment of the present disclosure, using the map ofregistered real cast shadows and at least one map of virtual shadows ofobjects in the 3D scene cast by the at least one light source from thevirtual reference viewpoint, comprises matching the map of registeredreal cast shadows and the at least one map of virtual shadows andselecting the map of virtual shadows that best matches the map ofregistered real cast shadows.

According to an embodiment of the present disclosure, the virtualreference viewpoint corresponds to a viewpoint from which a real scenerepresented by the 3D scene has been previously captured by the camera.

According to an embodiment of the present disclosure, the virtualreference viewpoint in the 3D scene is determined as being a viewpointfor which a number of pixels overlapping in an image captured from thevirtual reference viewpoint and an image captured from the viewpoint ofthe camera is above a determined value.

According to an embodiment of the present disclosure, when the virtualreference viewpoint and the camera's viewpoint do not satisfy adetermined criterion, a new virtual reference viewpoint is determined.

According to an embodiment of the present disclosure, the virtualreference viewpoint and the camera's viewpoint do not satisfy thedetermined criterion if a number of pixels overlapping in an imagecaptured from the virtual reference viewpoint and an image captured fromthe camera's viewpoint is below a determined value.

According to an embodiment of the present disclosure, the virtualreference viewpoint and the camera's viewpoint do not satisfy thedetermined criterion when a distance between the virtual referenceviewpoint and the camera's viewpoint is above a determined value.

According to an embodiment of the present disclosure, the method forprocessing a 3D scene, further comprises comparing the poses of thevirtual reference viewpoint and of the new virtual reference viewpointwith the camera poses, selecting among the virtual reference viewpointand of the new virtual reference viewpoint, the virtual referenceviewpoint closest to the camera's viewpoint, light source parameters inthe 3D scene being determined with regards to the selected virtualreference viewpoint.

According to this embodiment, a new virtual reference viewpoint ispre-computed, stored and selected when needed.

According to an embodiment of the present disclosure, determining saidat least one virtual reference viewpoint in the 3D scene, comprisesdetecting movement of the camera capturing the real scene, predicting acamera trajectory in the 3D scene as a function of the detectedmovement, said at least one virtual reference viewpoint being determinedaccording to the predicted camera trajectory.

According to this embodiment, a new virtual reference viewpoint can bedetermined in advance by anticipating the movement of the user, and ofthe camera. In this way, computations of the virtual shadow maps withrespect to the new virtual reference viewpoint can be carried out inadvance before the new virtual reference viewpoint is needed.

According to an embodiment of the present disclosure, determining saidat least one virtual reference viewpoint in the 3D scene comprisesdetermining and storing at least one shadow map associated to a lightsource of a 3D scene. According to this embodiment, virtual shadow mapscan be computed on the fly, before the virtual reference viewpoint towhich they are associated is needed, and stored for future use. In thatway, virtual shadow maps computation can be anticipated.

According to an embodiment of the present disclosure, the method forprocessing a 3D scene further comprises detecting geometry changes inthe 3D scene, determining areas in the input image captured by thecamera that are impacted by the geometry changes, said determined areasbeing discarded when determining parameters for at least one lightsource in the 3D scene using the virtual reference viewpoint untilvirtual shadow maps associated to the virtual reference viewpoint areupdated based on the geometry changes.

According to the embodiment, geometry changes can be taken into accountwhen determining lighting parameters.

One or more of the present embodiments also provide a computer readablestorage medium having stored thereon instructions for processing a 3Dscene, according to any one of the embodiments of the method describedabove. The present embodiments also provide a computer program productincluding instructions for performing any one of the embodiments of themethod described.

The implementations described herein may be implemented in, for example,a method or a process, an apparatus, a computer program product, a datastream, or a signal. Even if only discussed in the context of a singleform of implementation (for example, discussed only as a method or adevice), the implementation of features discussed may also beimplemented in other forms (for example a program). An apparatus may beimplemented in, for example, appropriate hardware, software, andfirmware. The methods may be implemented in, for example, an apparatussuch as, for example, a processor, which refers to processing devices ingeneral, including, for example, a computer, a microprocessor, anintegrated circuit, or a programmable logic device. Processors alsoinclude communication devices, such as, for example, Smartphones,tablets, computers, mobile phones, portable/personal digital assistants(“PDAs”), and other devices that facilitate communication of informationbetween end-users.

Implementations of the various processes and features described hereinmay be embodied in a variety of different equipment or applications,particularly, for example, equipment or applications associated withdata encoding, data decoding, view generation, texture processing, andother processing of images and related texture information and/or depthinformation. Examples of such equipment include an encoder, a decoder, apost-processor processing output from a decoder, a pre-processorproviding input to an encoder, a video coder, a video decoder, a videocodec, a web server, a set-top box, a laptop, a personal computer, acell phone, a PDA, and other communication devices. As should be clear,the equipment may be mobile and even installed in a mobile vehicle.

Additionally, the methods may be implemented by instructions beingperformed by a processor, and such instructions (and/or data valuesproduced by an implementation) may be stored on a processor-readablemedium such as, for example, an integrated circuit, a software carrieror other storage device such as, for example, a hard disk, a compactdiskette (“CD”), an optical disc (such as, for example, a DVD, oftenreferred to as a digital versatile disc or a digital video disc), arandom access memory (“RAM”), or a read-only memory (“ROM”). Theinstructions may form an application program tangibly embodied on aprocessor-readable medium. Instructions may be, for example, inhardware, firmware, software, or a combination. Instructions may befound in, for example, an operating system, a separate application, or acombination of the two. A processor may be characterized, therefore, as,for example, both a device configured to carry out a process and adevice that includes a processor-readable medium (such as a storagedevice) having instructions for carrying out a process. Further, aprocessor-readable medium may store, in addition to or in lieu ofinstructions, data values produced by an implementation.

As will be evident to one of skill in the art, implementations mayproduce a variety of signals formatted to carry information that may be,for example, stored or transmitted. The information may include, forexample, instructions for performing a method, or data produced by oneof the described implementations. For example, a signal may be formattedto carry as data the rules for writing or reading the syntax of adescribed embodiment, or to carry as data the actual syntax-valueswritten by a described embodiment. Such a signal may be formatted, forexample, as an electromagnetic wave (for example, using a radiofrequency portion of spectrum) or as a baseband signal. The formattingmay include, for example, encoding a data stream and modulating acarrier with the encoded data stream. The information that the signalcarries may be, for example, analog or digital information. The signalmay be transmitted over a variety of different wired or wireless links,as is known. The signal may be stored on a processor-readable medium.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. For example,elements of different implementations may be combined, supplemented,modified, or removed to produce other implementations. Additionally, oneof ordinary skill will understand that other structures and processesmay be substituted for those disclosed and the resulting implementationswill perform at least substantially the same function(s), in at leastsubstantially the same way(s), to achieve at least substantially thesame result(s) as the implementations disclosed. Accordingly, these andother implementations are contemplated by this application.

1. A method for processing a 3D scene, comprising: determining at leastone virtual reference viewpoint in the 3D scene, obtaining a map ofregistered real cast shadows of objects in the 3D scene from an inputimage captured by a camera positioned at a viewpoint distinct from thevirtual reference viewpoint, said map of real cast shadows beingregistered with regards to the virtual reference viewpoint, anddetermining parameters for at least one light source in the 3D sceneusing the map of registered real cast shadows and at least one map ofvirtual shadows of objects in the 3D scene cast by the at least onelight source from the virtual reference viewpoint.
 2. An apparatus forprocessing a 3D scene comprising: means for determining at least onevirtual reference viewpoint in the 3D scene, means for obtaining a mapof registered real cast shadows of objects in the 3D scene from an inputimage captured by the camera positioned at a viewpoint distinct from thevirtual reference viewpoint, said map of real cast shadows beingregistered with regards to the virtual reference viewpoint, and meansfor determining parameters for at least one light source in the 3D sceneusing the map of registered real cast shadows and at least one map ofvirtual shadows of objects in the 3D scene cast by the at least onelight source from the virtual reference viewpoint.
 3. The methodaccording to claim 1, wherein obtaining the map of registered real castshadows comprises: detecting the real cast shadows in the input image toobtain a map of detected real cast shadows, and registering the map ofdetected real cast shadows with regards to the virtual referenceviewpoint.
 4. The method according to claim 1, wherein obtaining the mapof registered real cast shadows comprises: registering the input imagewith regards to the virtual reference viewpoint to obtain a registeredimage, and detecting the real cast shadows in the registered image. 5.The method according to claim 1, wherein using the map of registeredreal cast shadows and at least one map of virtual shadows of objects inthe 3D scene cast by the at least one light source from the virtualreference viewpoint, comprises matching the map of registered real castshadows and the at least one map of virtual shadows and selecting themap of virtual shadows that best matches the map of registered real castshadows.
 6. The method according to claim 1, wherein the virtualreference viewpoint corresponds to a viewpoint from which a real scenerepresented by the 3D scene has been previously captured by the camera.7. The method according to claim 1, wherein the virtual referenceviewpoint in the 3D scene is determined as being a viewpoint for which anumber of pixels overlapping in an image captured from the virtualreference viewpoint and an image captured from the viewpoint of thecamera is above a determined value.
 8. The method according to claim 1,wherein when the virtual reference viewpoint and the camera's viewpointdo not satisfy a determined criterion, a new virtual reference viewpointis determined.
 9. The method according to claim 8, wherein the virtualreference viewpoint and the camera's viewpoint do not satisfy thedetermined criterion if a number of pixels overlapping in an imagecaptured from the virtual reference viewpoint and an image captured fromthe camera's viewpoint is below a determined value.
 10. The methodaccording to claim 8, further comprising: comparing the poses of thevirtual reference viewpoint and of the new virtual reference viewpointwith the camera poses, and selecting among the virtual referenceviewpoint and the new virtual reference viewpoint, the virtual referenceviewpoint closest to the camera's viewpoint, light source parameters inthe 3D scene being determined with regards to the selected virtualreference viewpoint.
 11. The method according to claim 1, whereindetermining said at least one virtual reference viewpoint in the 3Dscene, comprises: detecting movement of the camera capturing the realscene, and predicting a camera trajectory in the 3D scene as a functionof the detected movement, said at least one virtual reference viewpointbeing determined according to the predicted camera trajectory.
 12. Themethod according to claim 1, wherein determining said at least onevirtual reference viewpoint in the 3D scene comprises determining andstoring at least one shadow map associated to a light source of a 3Dscene.
 13. The method according to claim 1, further comprising:detecting geometry changes in the 3D scene, and determining areas in theinput image captured by the camera that are impacted by the geometrychanges, said determined areas being discarded when determiningparameters for at least one light source in the 3D scene using thevirtual reference viewpoint until virtual shadow maps associated to thevirtual reference viewpoint are updated based on the geometry changes.14. (canceled)
 15. The apparatus according to claim 2, wherein obtainingthe map of registered real cast shadows comprises: detecting the realcast shadows in the input image to obtain a map of detected real castshadows, and registering the map of detected real cast shadows withregards to the virtual reference viewpoint.
 16. The apparatus accordingto claim 2, wherein obtaining the map of registered real cast shadowscomprises: registering the input image with regards to the virtualreference viewpoint to obtain a registered image, and detecting the realcast shadows in the registered image.
 17. The apparatus according toclaim 2, wherein using the map of registered real cast shadows and atleast one map of virtual shadows of objects in the 3D scene cast by theat least one light source from the virtual reference viewpoint,comprises matching the map of registered real cast shadows and the atleast one map of virtual shadows and selecting the map of virtualshadows that best matches the map of registered real cast shadows. 18.The apparatus according to claim 2, wherein the virtual referenceviewpoint corresponds to a viewpoint from which a real scene representedby the 3D scene has been previously captured by the camera.
 19. Theapparatus according to claim 2, wherein the virtual reference viewpointin the 3D scene is determined as being a viewpoint for which a number ofpixels overlapping in an image captured from the virtual referenceviewpoint and an image captured from the viewpoint of the camera isabove a determined value.
 20. A non-transitory computer-readable mediumstoring instructions operative, when executed by a processor, to performat least: determining at least one virtual reference viewpoint in the 3Dscene, obtaining a map of registered real cast shadows of objects in the3D scene from an input image captured by a camera positioned at aviewpoint distinct from the virtual reference viewpoint, said map ofreal cast shadows being registered with regards to the virtual referenceviewpoint, and determining parameters for at least one light source inthe 3D scene using the map of registered real cast shadows and at leastone map of virtual shadows of objects in the 3D scene cast by the atleast one light source from the virtual reference viewpoint.